A Machine Learning Approach to Recovery of Scene Geometry from Images

Recovering the 3D structure of the scene from images yields useful information for tasks such as shape and scene recognition, object detection, or motion planning and object grasping in robotics. In this thesis, we introduce a general machine learning approach called unsupervised CRF learning based on maximizing the conditional likelihood. We describe the application of our machine learning approach to computer vision systems that recover the 3-D scene geometry from images. We focus on recovering 3D geometry from single images, stereo pairs and video sequences. Building these systems requires algorithms for doing inference as well as learning the parameters of conditional Markov random fields (MRF). Unlike previous work, our system is trained unsupervisedly without using ground-truth labeled data. We employ a slanted-plane stereo vision model in which we use a fixed over-segmentation to segment the left image into coherent regions called superpixels. We then assign a disparity plane for each superpixel. We formulate the problem of inferring plane parameters as an MRF labelling problem, which can be solved by an energy minimization method. The MRF is a graphical model in which superpixels define nodes and the adjacency between superpixels define edges. Our stereo energy function balances between a data matching term and a smoothness term. For systems with continuous valued variables, or discrete-valued variables with very large state space, it is impossible to directly use a standard discrete MRF inference techniques such as Loopy BP, graph cuts or tree-reweighted message passing. For such systems, we propose to use a generic Particle-based Belief Propagation (PBP) algorithm closely related to previous work, which we then formulate specifically for our MRF labeling problems. Although we only describe a specific use of this generic PBP algorithm, we believe it can be used as an approximate inference scheme for a wide variety of problems that can be formulated by a probabilistic graphical model, especially those containing many random variables with very large or continuous domains. We demonstrate the use of our unsupervised CRF learning algorithm for a parameterized slanted-plane stereo vision model involving shape from texture cues. This unsupervised learning algorithm implicitly trains shape from texture monocular surface orientation iii iv cues. We exhibit that training monocular cues from stereo pair data improves stereo depth estimation. Our stereo model with texture cues, only by unsupervised training, outperform the results in related work on the same stereo dataset. Our unsupervised learning method is also implemented for the monocular depth estimation (MDE) problem. The MDE model, learned using stereo pairs only, demonstrates a modest improvement after a few training steps, and achieve performance comparable to previous work on the same dataset. The use of MDE in combination with the dense stereo model also introduces a small boost in depth estimation over the initial stereo model. In this thesis, we also address the use of stereo video sequences. We formulate structure and motion estimation as an energy minimization problem, in which the model is an extension of our slanted-plane stereo vision model that also handles surface velocity. Surface estimation is done using our own slanted-plane stereo algorithm. Velocity estimation is achieved by solving an MRF labeling problem using Loopy BP. Performance analysis is done using our novel evaluation metrics based on the notion of view prediction error. Experiments on road-driving stereo sequences show encouraging results.